Monado Session状态流转

发表于   |   分类于   |   6,132 字   |   28 分钟   |  

Monado Session状态流转

先前在《Monado Out of Process流程分析》一文中有提到了XrSession的部分，只是简单描述了一下整个XrSession的创建流程，对于OpenXR来说，XrSession是类似于Android中Activity的存在，是一个比较重要的概念，因此打算特别写一篇文档来讲述一下XrSession的几个状态流转过程和对应的具体场景。

在OpenXR的spec中，XrSession是有定义的：

A session represents an application’s intention to display XR content to the user.

需要速成的可以直接到最后看时序图《Flow Chart》

Life Cycle

官方Spec的定义中，一共给出了8个状态，我把这8个状态分为了三种类型，简单称为：初始，可见，退出。

  • 初始：IDLE，READY，SYNCHRONIZED
  • 可见：VISIBLE，FOCUS
  • 退出：STOPPING，EXITING，LOSS_PENDING

以下是Spec附上的XrSession Life Cycle流转图：

img

State Defined

各个State的详细定义在Spec上是有专门介绍的，这里我们逐条理解一下。

A typical XR session coordinates the application and the runtime through session control functions and session state events.

  • OpenXR Session的state event是跟随着Session相关的控制函数调用而改变的（等于没说）
  1. The application creates a session by choosing a system and a graphics API and passing them into xrCreateSession. The newly created session is in the XR_SESSION_STATE_IDLE state.
  • 应用在通过XrCreateSession创建Session的时候，需要带入System和Graphic API，其中System ID是在XrCreateInstance后可以通过XrGetSystem来获取，而Graphic API的部分，以OpenGLES为例，就是填充它：XrGraphicsBindingOpenGLESAndroidKHR
  • 新创建出来的Session是处于XR_SESSION_STATE_IDLE状态
  1. The application then monitors for session state changes via XrEventDataSessionStateChanged events.
  • 在Session创建完成之后，应用程序可以通过监听xrPollEvent中type为XR_TYPE_EVENT_DATA_SESSION_STATE_CHANGED的事件来知晓Session状态的变化
  1. When the runtime determines that the system is ready to start transitioning to this session’s XR content, the application receives a notification of session state change to XR_SESSION_STATE_READY. Once the application is also ready to proceed and display its XR content, it calls xrBeginSession and starts its frame loop, which begins a running session.
  • 这段文字就非常奥妙了，它表达了两层含义：
    • 第一层含义是，如果Runtime的System已经ready了，并可以开始处理，那么应用端应该会收到XR_SESSION_STATE_READY的通知，显然，这个通知应该由Runtime负责发出来。
    • 第二层含义是，如果应用端已经准备生产（画东西）和显示XR的内容时，那么它需要调用xrBeginSession并开始做frame loop，通过调用xrBegionSession，然后把这个Session视为a running session
  1. While the session is running, the application is expected to continuously execute its frame loop by calling xrWaitFrame, xrBeginFrame and xrEndFrame each frame, establishing synchronization with the runtime. Once the runtime is synchronized with the application’s frame loop and ready to display application’s frames, the session moves into the XR_SESSION_STATE_SYNCHRONIZED state. In this state, the submitted frames will not be displayed or visible to the user yet.
  • 在Session进入到running状态后，应用就可以开启xrWaitFrame，xrBeginFrame，xrEndFrame的循环了，通过这个流程跟Runtime端建立synchronization机制，就是一个frame的生产/消费同步。
  • 当这个同步机制建立以后，那就意味着Runtime端已经做好了显示应用frame的准备了，这个时候Session需要切换到XR_SESSION_STATE_SYNCHRONIZED的状态。
  • 同时，当Session处于XR_SESSION_STATE_SYNCHRONIZED状态时，应用提交的frame是不会被显示出来的。
  1. When the runtime intends to display frames from the application, it notifies with XR_SESSION_STATE_VISIBLE state, and sets XrFrameState::shouldRender to true in xrWaitFrame. The application should render XR content and submit the composition layers to xrEndFrame.
  • 当Runtime已经准备好显示来自应用的Frame时，就可以把Session的状态切换到XR_SESSION_STATE_VISIBLE。
  • 同时，当Session处于XR_SESSION_STATE_VISIBLE的状态时，应用在调用xrWaitFrame时的，其获取的返回值：XrFrameState::shouldRender为True
  • 对于应用来说，需要绘制的所有内容会以layer的形式一并提交给Session，也就是当应用调用xrEndFrane的时候，Runtime那边才会收到来自应用的绘制内容。
  1. When the runtime determines the application is eligible to receive XR inputs, e.g. motion controller or hand tracking inputs, it notifies with XR_SESSION_STATE_FOCUSED state. The application can expect to receive active action inputs.
  • Runtime会判断哪个应用可以收到来自XR设备的input事件，这个时候就会发送XR_SESSION_STATE_FOCUSED，从这之后，应用就可以期望收到action input的事件了。
    • 这一段描述中，可以看到理论上只有一个Session可以处于Focus的状态了，望文生义determines the application is而不是applications?
  1. When the runtime determines the application has lost XR input focus, it moves the session state from XR_SESSION_STATE_FOCUSED to XR_SESSION_STATE_VISIBLE state. The application may need to change its own internal state while input is unavailable. Since the session is still visible, the application needs to render and submit frames at full frame rate, but may wish to change visually to indicate its input suspended state. When the runtime returns XR focus back to the application, it moves the session state back to XR_SESSION_STATE_FOCUSED.
  • Runtime会根据一些情况来判断当前application是否可以继续收到input事件，如果丢失了input focus，那么就会从XR_SESSION_STATE_FOCUSED切换到XR_SESSION_STATE_VISIBLE，这个时候spec的建议是应用需要自行决断，做一些适当的业务行为的变更。同时，由于这个时候应用的Session实际还是处于visible的状态，因此还是要保证full frame rate。
  • 如果Runtime在后续的行为中，把focus交还给了这个应用，那么还是同样需要把Session的状态切回到XR_SESSION_STATE_FOCUSED的状态。
  1. When the runtime needs to end a running session due to the user closing or switching the application, the runtime will change the session state through appropriate intermediate ones and finally to XR_SESSION_STATE_STOPPING. When the application receives the XR_SESSION_STATE_STOPPING event, it should stop its frame loop and then call xrEndSession to tell the runtime to stop the running session.
  • 当Runtime因为收到来自用户的退出行为，或者是需要切换到其他应用时，就需要切换到XR_SESSION_STATE_STOPPING的状态
  • 当应用接收到来自Runtime的XR_SESSION_STATE_STOPPING通知时，需要停掉它的frame loop并调用xrEndSession，以告知Runtime当前Session需要stop。
  1. After xrEndSession, the runtime transitions the session state to XR_SESSION_STATE_IDLE. If the XR session is temporarily paused in the background, the runtime will keep the session state at XR_SESSION_STATE_IDLE and later transition the session state back to XR_SESSION_STATE_READY when the XR session is resumed. If the runtime determines that its use of this XR session has concluded, it will transition the session state from XR_SESSION_STATE_IDLE to XR_SESSION_STATE_EXITING.
  • 当应用端调用了xrEndSession之后，Runtime会把当前Session的状态变更为XR_SESSION_STATE_IDLE。如果只是临时的pasue，那么后面Runtime会继续保持Session为XR_SESSION_STATE_IDLE的状态，并在适当时候重新切到XR_SESSION_STATE_READY的状态。
  • 如果Runtime认为当前的Session已经是完全结束了，那么就会把它的状态从XR_SESSION_STATE_IDLE切换到XR_SESSION_STATE_EXITING
  1. When the application receives the XR_SESSION_STATE_EXITING event, it releases the resources related to the session and calls xrDestroySession.
  • 当应用收到XR_SESSION_STATE_EXITING的事件后，需要调用xrDestroySession来完成整个闭环

Code Impl

Help Func

在看整个Session流转的flow之前，我们需要先看一下对应的push和poll是怎么做的。

State Push

在函数调用上，是直接通过oxr_session_change_state来开启整个流程的。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
void
oxr_session_change_state(struct oxr_logger *log, struct oxr_session *sess, XrSessionState state)
{
    ......
    // 1. 最后的参数为0，实际是XrTime，也就是对于session change而言，在monado方案中time不重要
	oxr_event_push_XrEventDataSessionStateChanged(log, sess, state, 0);
    // 2. session实际状态的改变发生在event入队列以后
	sess->state = state;
}
XrResult
oxr_event_push_XrEventDataSessionStateChanged(struct oxr_logger *log,
                                              struct oxr_session *sess,
                                              XrSessionState state,
                                              XrTime time)
{
	struct oxr_instance *inst = sess->sys->inst;
	XrEventDataSessionStateChanged *changed;
	struct oxr_event *event = NULL;
    // 1. 构造changed元素，这里的ALLOC是个宏
	ALLOC(log, inst, &event, &changed);
    // 实际ALLOC宏展开后：
    // XrResult ret = oxr_event_alloc(log, inst, sizeof(**(&changed)), event);
    // if (ret != XR_SUCCESS) {
    //     return ret;
    // }
    // *((void **)(&changed)) = oxr_event_extra(*event);
   
    // 2. 其中 oxr_event_alloc
    // static XrResult
    // oxr_event_alloc(struct oxr_logger *log, struct oxr_instance *inst, size_t size, struct oxr_event **out_event)
    // {
    //     struct oxr_event *event = U_CALLOC_WITH_CAST(struct oxr_event, sizeof(struct oxr_event) + size);
    //     if (event == NULL) {
    //         return oxr_error(log, XR_ERROR_RUNTIME_FAILURE, "Out of memory");
    //     }
    //     event->next = NULL;
    //     event->length = size;
    //     event->result = XR_SUCCESS;
    //     *out_event = event;
    //     return XR_SUCCESS;
    // }
    // 3. 再看oxr_event_extra，实际就是跳过sizeof(struct oxr_event)，获取下一个地址
    // void *
	// oxr_event_extra(struct oxr_event *event)
	// {
	//     return &event[1];
	// }
    // *((void **)(&changed)) = oxr_event_extra(*event);
    
    // 4. 通过上面这一段的操作，实际上我们完成了对event的内存malloc和它之后XrEventDataSessionStateChanged bit位的初始化
    // |  struct oxr_event               |  -> all begin with : event
    // |  XrEventDataSessionStateChanged |  -> 
    // event = addressA
    // changed = addressA + sizeof(struct oxr_event)
    
    // 5. 有了以上的认知，我们就可以知道如下的事实：
    //    如果拿到了event，其实也就是拿到了changed。
	changed->type = XR_TYPE_EVENT_DATA_SESSION_STATE_CHANGED;
	changed->session = oxr_session_to_openxr(sess);
	changed->state = state;
	changed->time = time;
	event->result = state == XR_SESSION_STATE_LOSS_PENDING ? XR_SESSION_LOSS_PENDING : XR_SUCCESS;
	......
	lock(inst);
    // 6. 因此，我们只需要处理event就可以了
	push(inst, event);
	unlock(inst);
	......
	return XR_SUCCESS;
}

而push函数则比较单纯了：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
//push(inst, event);
void
push(struct oxr_instance *inst, struct oxr_event *event)
{
    // 1. 这个就是典型的队列事件
	struct oxr_event *last = inst->event.last;
    // 2. 第一个事件的last为null
	if (last != NULL) {
		last->next = event;
	}
    // 3. 此时把 inst->event.last指向为新来的event
	inst->event.last = event;
	// 4. 由于这里是第一个事件，因此next为null
	if (inst->event.next == NULL) {
		inst->event.next = event;
	}
    // 5. 完成了next和last指针的赋值
}

State Pop

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
XrResult
oxr_poll_event(struct oxr_logger *log, struct oxr_instance *inst, XrEventDataBuffer *eventData)
{
	struct oxr_session *sess = inst->sessions;
    // 1. 这个函数在后面分析，需要从Runtime进程去获取event，实际是在根据Runtime进程的情况，在application进程做状态切换。
	while (sess) {
		oxr_session_poll(log, sess);
		sess = sess->next;
	}
	
	lock(inst);
    // 2.pop，获取到队列头的event
	struct oxr_event *event = pop(inst);
	unlock(inst);
	......
	XrResult ret = event->result;
    // 这里要记得在push章节分析过的oxr_event_extra函数，实际是return &event[1];
    // 因此这里的memcpy的实际是把changed数据拿了出来，放到eventData
    // 也就是push时候的：XrEventDataSessionStateChanged *changed;
    // 3.拿到event数据
	memcpy(eventData, oxr_event_extra(event), event->length);
	free(event);
	return ret;
}
// 2.pop
struct oxr_event *
pop(struct oxr_instance *inst)
{
    // 1. pop的实现也是典型的队列形式，首先是获取到next指针
	struct oxr_event *ret = inst->event.next;
	if (ret == NULL) {
		return NULL;
	}
    // 2. 当next指针不为null，那就是说当前有数据需要处理，直接拿出来
	inst->event.next = ret->next;
	ret->next = NULL;
    // 3.如果当前的next已经是last，也即最后一个数据，那么把队尾指针也设置为null
	if (ret == inst->event.last) {
		inst->event.last = NULL;
	}
	return ret;
}

Application Code Flow

以HelloXR的代码为例，描述一下整个应用端对于Session的是怎么做流转的。

android main loop

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
void android_main(struct android_app* app) {
        program->CreateInstance();
        program->InitializeSystem();
        // 1. 初始化session
        program->InitializeSession();
        program->CreateSwapchains();
        while (app->destroyRequested == 0) {
            ......
            // 2. 获取各种events
            program->PollEvents(&exitRenderLoop, &requestRestart);
            if (!program->IsSessionRunning()) {
                // Throttle loop since xrWaitFrame won't be called.
                std::this_thread::sleep_for(std::chrono::milliseconds(250));
                continue;
            }
			// 3. 获取action
            program->PollActions();
            // 4. 绘制
            program->RenderFrame();
        }
}

init session

1
2
3
4
5
6
7
program->InitializeSession();
	void InitializeSession() override {
        ......
        // 1. 这里会调用到xrCreateSession
        CHECK_XRCMD(xrCreateSession(m_instance, &createInfo, &m_session));
        ......
    }

Generate IDLE -> READY

IDLE和READY的事件是在xrCreateSession的时候push进去的。

  • xrCreateSession
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
XrResult
oxr_xrCreateSession(XrInstance instance, const XrSessionCreateInfo *createInfo, XrSession *out_session) {
	......
    ret = oxr_session_create(&log, &inst->system, createInfo, &sess);
    ......
}
XrResult
oxr_session_create(struct oxr_logger *log,
                   struct oxr_system *sys,
                   const XrSessionCreateInfo *createInfo,
                   struct oxr_session **out_session)
{
	......
    // 1. 根据spec的定义，在create session的时候需要置为IDLE状态
	oxr_session_change_state(log, sess, XR_SESSION_STATE_IDLE);
    // 2. Monado的方案中，create session完成以后就认为当前sessin是ready的了，所以两个事件在createSession中一起push了
	oxr_session_change_state(log, sess, XR_SESSION_STATE_READY);
	......
	return ret;
}

xr events loop

  • PollEvents函数
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
// 2. 获取各种events
program->PollEvents(&exitRenderLoop, &requestRestart);
    void PollEvents(bool* exitRenderLoop, bool* requestRestart) override {
        *exitRenderLoop = *requestRestart = false;
        // Process all pending messages.
        // 1. 调用TryReadNextEvent，这个函数实际是xrPollEvent
        while (const XrEventDataBaseHeader* event = TryReadNextEvent()) {
            switch (event->type) {
                case XR_TYPE_EVENT_DATA_INSTANCE_LOSS_PENDING: {
                    const auto& instanceLossPending = *reinterpret_cast<const XrEventDataInstanceLossPending*>(event);
                    ......
                    *exitRenderLoop = true;
                    *requestRestart = true;
                    return;
                }
                case XR_TYPE_EVENT_DATA_SESSION_STATE_CHANGED: {
                    auto sessionStateChangedEvent = *reinterpret_cast<const XrEventDataSessionStateChanged*>(event);
                    // 2. 对于是XR_TYPE_EVENT_DATA_SESSION_STATE_CHANGED的事件
                    //    会继续跑一段业务代码HandleSessionStateChangedEvent
                    HandleSessionStateChangedEvent(sessionStateChangedEvent, exitRenderLoop, requestRestart);
                    break;
                }
                ......
            }
        }
    }
 	// 1. 调用TryReadNextEvent，这个函数实际是xrPollEvent
	// while (const XrEventDataBaseHeader* event = TryReadNextEvent()) {
    const XrEventDataBaseHeader* TryReadNextEvent() {
        ......
        const XrResult xr = xrPollEvent(m_instance, &m_eventDataBuffer);
        if (xr == XR_SUCCESS) {
            if (baseHeader->type == XR_TYPE_EVENT_DATA_EVENTS_LOST) {
                const XrEventDataEventsLost* const eventsLost = reinterpret_cast<const XrEventDataEventsLost*>(baseHeader);
                ......
            }
            return baseHeader;
        }
        ....
    }
    // 2. 对于是XR_TYPE_EVENT_DATA_SESSION_STATE_CHANGED的事件
    //    会继续跑一段业务代码HandleSessionStateChangedEvent
	// HandleSessionStateChangedEvent(sessionStateChangedEvent, exitRenderLoop, requestRestart);
    void HandleSessionStateChangedEvent(const XrEventDataSessionStateChanged& stateChangedEvent, bool* exitRenderLoop,
                                        bool* requestRestart) {
        ......
        m_sessionState = stateChangedEvent.state;
        ......
        switch (m_sessionState) {
            // 1. 当接收到READY的状态后，应用端会开始做xrBeginSessin的操作，并置m_sessionRunning=true
            //    这样的做法，实际也是暗合了spec中第3条后半段的说法
            case XR_SESSION_STATE_READY: {
                ......
                CHECK_XRCMD(xrBeginSession(m_session, &sessionBeginInfo));
                m_sessionRunning = true;
                break;
            }
            // 2. 当接收到STOPPING的状态后，应用端会开始做xrEndSession的操作，并置m_sessionRunning=false
            //    这样的做法，实际也是暗合了spec中第8条的说法      
            case XR_SESSION_STATE_STOPPING: {
                ......
                m_sessionRunning = false;
                CHECK_XRCMD(xrEndSession(m_session))
                break;
            }
            case XR_SESSION_STATE_EXITING: {
                *exitRenderLoop = true;
                // Do not attempt to restart because user closed this session.
                *requestRestart = false;
                break;
            }
            case XR_SESSION_STATE_LOSS_PENDING: {
                *exitRenderLoop = true;
                // Poll for a new instance.
                *requestRestart = true;
                break;
            }
            default:
                break;
        }
    }

render loop

1
2
3
4
5
6
7
8
9
10
// 4. 绘制
program->RenderFrame();
    void RenderFrame() override {
        ......
        CHECK_XRCMD(xrWaitFrame(m_session, &frameWaitInfo, &frameState));
        ......
        CHECK_XRCMD(xrBeginFrame(m_session, &frameBeginInfo));
        ......
        CHECK_XRCMD(xrEndFrame(m_session, &frameEndInfo));
    }

From IDLE -> READY

  • 当应用端第一次调用xrCreateSession的时候，在Runtime Client会直接push两个事件到event队列：

    • XR_SESSION_STATE_IDLE
    • XR_SESSION_STATE_READY

  • 在应用进行第一次program->PollEvents(&exitRenderLoop, &requestRestart);的时候，会拿到XR_SESSION_STATE_IDLE

  • 由于在PollEvents中是循环调用的xrPolLEvents，因此在处理完IDLE事件后，会马上收到：XR_SESSION_STATE_READY
  • 这个时候会进入到Session Changed业务函数：HandleSessionStateChangedEvent
    • 调用xrBeginSession
    • 并把并置m_sessionRunning=true
  • 这个时候应用端就会进入到render loop中，调用到xrWaitFrame

至此，整个流程已经走到了READY的状态，那么根据spec第4步的描述，需要建议SYNC的机制：

  1. While the session is running, the application is expected to continuously execute its frame loop by calling xrWaitFrame, xrBeginFrame and xrEndFrame each frame, establishing synchronization with the runtime. Once the runtime is synchronized with the application’s frame loop and ready to display application’s frames, the session moves into the XR_SESSION_STATE_SYNCHRONIZED state. In this state, the submitted frames will not be displayed or visible to the user yet.
  • 在Session进入到running状态后，应用就可以开启xrWaitFrame，xrBeginFrame，xrEndFrame的循环了，通过这个流程跟Runtime端建立synchronization机制，就是一个frame的生产/消费同步。
  • 当这个同步机制建立以后，那就意味着Runtime端已经做好了显示应用frame的准备了，这个时候Session需要切换到XR_SESSION_STATE_SYNCHRONIZED的状态。
  • 同时，当Session处于XR_SESSION_STATE_SYNCHRONIZED状态时，应用提交的frame是不会被显示出来的。

From READY -> SYNC

当进入到第一个render loop，我们按照正常的流程运行xrWaitFrame,xrBeginFrame,xrEndFrame。

  • 由于我们当前的状态为READY，所以在应用调用xrWaitFrame的时候shouldRender=False。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
XrResult
oxr_session_frame_wait(struct oxr_logger *log, struct oxr_session *sess, XrFrameState *frameState)
{
	.......
    // 1. wait frame的时候会触发一次ipc操作，这边就不做展开了，简单来说在这个ipc中会调用到
    //    Runtime进程，并最终调用到system_compositor_set_state，并设置起client端对应的。
    //    具体的流程可以参考《Monado Out Of Process流程分析》一文的《Frame-xrWaitFrame-Impl Code-In RPC》章节
    //    传入的参数，由于ics->server_thread_index == active_id，因此：visible = true，focused = true;
    //          xce.type = XRT_COMPOSITOR_EVENT_STATE_CHANGE;
    //    		xce.state.visible = visible;
	//        	xce.state.focused = focused;
    //    最后会push一个multi_event到队尾，这个部分就是在oxr_poll_event函数中提到但是没有分析的：oxr_session_poll
    //	  这部分的事件是来源于Runtime进程的，所以当第一次调用xrWaitFrame的时候，实际上会产生一个XRT_COMPOSITOR_EVENT_STATE_CHANGE事件
        
    CALL_CHK(xrt_comp_wait_frame(xc, &sess->frame_id.waited, &predicted_display_time, &predicted_display_period));
	......
    // 2. 这里会根据session的状态来赋：frameState->shouldRender的值
    frameState->shouldRender = should_render(sess->state);
    ......
}
static bool
should_render(XrSessionState state)
{
	switch (state) {
    	// 1. 仅有三种状态下shouldRender的值为true
        case XR_SESSION_STATE_VISIBLE: return true;
        case XR_SESSION_STATE_FOCUSED: return true;
        case XR_SESSION_STATE_STOPPING: return true;
        default: return false;
	}
}
  • 因此在应用端进行xrBeginFrame和xrEndFrame时，会有一次空layer的情况。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
void RenderFrame() override {
    ......
    // 1. 这里调用xrWaitFrame，但是由于session是READY的状态，因此shouldRender=false
    CHECK_XRCMD(xrWaitFrame(m_session, &frameWaitInfo, &frameState));
    ......
    // 2. 因为shouldRender=false，所以就不会进入到render layer的流程
    if (frameState.shouldRender == XR_TRUE) {
        if (RenderLayer(frameState.predictedDisplayTime, projectionLayerViews, layer)) {
            layers.push_back(reinterpret_cast<XrCompositionLayerBaseHeader*>(&layer));
        }
    }
    // 3. 进行xrEndFrame的时候，可以看到layer size = 0
    ......
    frameEndInfo.layerCount = (uint32_t)layers.size();
    frameEndInfo.layers = layers.data();
    CHECK_XRCMD(xrEndFrame(m_session, &frameEndInfo));
}
  • 跑到Runtime Client端在做xrEndFrame时，会因为layer size = 0进入到一个特殊的分支frameEndInfo->layerCount == 0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
XrResult
oxr_session_frame_end(struct oxr_logger *log, struct oxr_session *sess, const XrFrameEndInfo *frameEndInfo)
{
	......
	if (frameEndInfo->layerCount == 0) {
		......
		do_synchronize_state_change(log, sess);
		return oxr_session_success_result(sess);
	}
  ......
}
static void
do_synchronize_state_change(struct oxr_logger *log, struct oxr_session *sess)
{
	if (!sess->has_ended_once && sess->state < XR_SESSION_STATE_VISIBLE) {
		oxr_session_change_state(log, sess, XR_SESSION_STATE_SYNCHRONIZED);
		sess->has_ended_once = true;
	}
}
  • 从而完成了一个XR_SESSION_STATE_SYNCHRONIZED的事件入队，这个也是符合spec第4步的定义：
  • Once the runtime is synchronized with the application’s frame loop and ready to display application’s frames, the session moves into the XR_SESSION_STATE_SYNCHRONIZED state. In this state, the submitted frames will not be displayed or visible to the user yet.

From SYNC -> VISIBLE -> FOCUS

由于在空Layer的xrEndFrame中产生了一个XR_SESSION_STATE_SYNCHRONIZED，因此在下一个轮询，也就是第三次：program->PollEvents(&exitRenderLoop, &requestRestart);的时候：

  • 应用端会收到：XR_SESSION_STATE_SYNCHRONIZED，但是在业务流程中并没有涉及，因此不做处理
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
    void PollEvents(bool* exitRenderLoop, bool* requestRestart) override {
        ......
        // 1. 这里依旧是轮询所有的事件
        while (const XrEventDataBaseHeader* event = TryReadNextEvent()) {
            ......
        }
    }
    const XrEventDataBaseHeader* TryReadNextEvent() {
        ......
        // 2. 实际是调用xrPollEvent
        const XrResult xr = xrPollEvent(m_instance, &m_eventDataBuffer);
        ......
    }
=> To Runtime Client
XrResult
oxr_poll_event(struct oxr_logger *log, struct oxr_instance *inst, XrEventDataBuffer *eventData)
{
	struct oxr_session *sess = inst->sessions;
	while (sess) {
        // 3. 这里会遍历所有的session，依次去看session的事件
		oxr_session_poll(log, sess);
		sess = sess->next;
	}
	......
}
void
oxr_session_poll(struct oxr_logger *log, struct oxr_session *sess)
{
	......
	bool read_more_events = true;
	while (read_more_events) {
		union xrt_compositor_event xce = {0};
        // 4. 这里会触发一次IPC调用，实际就会去查看Runtime进程中的事件队列，
        //    在这里其实就会拿到 《From READY -> SYNC》章节中提到的XRT_COMPOSITOR_EVENT_STATE_CHANGE事件
		xc->poll_events(xc, &xce);
		// dispatch based on event type
		switch (xce.type) {
		......
		// 5. 获取到XRT_COMPOSITOR_EVENT_STATE_CHANGE，然后把sess->compositor_visible
        //    sess->compositor_visible
		case XRT_COMPOSITOR_EVENT_STATE_CHANGE:
			sess->compositor_visible = xce.state.visible;
			sess->compositor_focused = xce.state.focused;
			break;
		......
		}
	}
    // 6. 因为当前Session已经是处于SYNC，visible也是true，直接进入if判断内
	if (sess->state == XR_SESSION_STATE_SYNCHRONIZED && sess->compositor_visible) {
        // 7. 设置当前Session为VISIBLE，同时产生VISIBLE事件
		oxr_session_change_state(log, sess, XR_SESSION_STATE_VISIBLE);
	}
    
	// 8. 因为当前Session已经是处于VISIBLE，focus也是true，直接进入if判断内
	if (sess->state == XR_SESSION_STATE_VISIBLE && sess->compositor_focused) {
        // 9. 设置当前Session为FOCUSED，同时产生FOCUSED事件
		oxr_session_change_state(log, sess, XR_SESSION_STATE_FOCUSED);
	}
}

因此在这个poll的轮询中，我们又push了两个事件：XR_SESSION_STATE_VISIBLE，XR_SESSION_STATE_FOCUSED。

所以应用端也就会马上响应到这两个事件了。

  • 在代码中，oxr_session_poll的似乎存在一个bug，就是无法从FOCUS状态回退到VISIBLE状态，加入我们出现了VISIBLE为true，但是FOCUSE为false的情况，当XRT_COMPOSITOR_EVENT_STATE_CHANGE事件发生时：
1
2
3
4
5
6
7
8
9
   // 1. 当前state为FOCUS，但是visible为True，因此不进入if内
if (sess->state == XR_SESSION_STATE_SYNCHRONIZED && sess->compositor_visible) {
	oxr_session_change_state(log, sess, XR_SESSION_STATE_VISIBLE);
}
   
// 2. 当前state为FOCUS，但是FOCUS为false，因此不进入if内
if (sess->state == XR_SESSION_STATE_VISIBLE && sess->compositor_focused) {
	oxr_session_change_state(log, sess, XR_SESSION_STATE_FOCUSED);
}

  • 而实际情况，在Runtime进程端：

    • ics->client_state.session_visible = visible;
    • ics->client_state.session_focused = focused;

    永远是相同的情况，也就是永远为ture或者永远为false。

这一条跟Spec中的第7步其实是有出入的，后面也可能是我们需要去额外实现的功能。

之后的步骤实际是从FOCUS的回退了。

From FOCUS -> … -> EXITING

在HelloXr的Sample Code中，需要通过用户按下quit键来实现退出的功能，但实际在手机端进行操作时由于Action章节还没有完全看明白，所以无法接上这一块的代码，只能通过修改一下代码，做patch方案。

1
2
3
4
5
6
7
8
9
10
void PollActions() override {
    static int i_patchCount = 0;
    i_patchCount++;
    ......
    CHECK_XRCMD(xrGetActionStateBoolean(m_session, &getInfo, &quitValue));
    // 1. 每次轮询的时候i_patchCount会自增，当i_patchCount==100时，假装触发了一次quit按键
    if ((i_patchCount == 100) ||......) {
        CHECK_XRCMD(xrRequestExitSession(m_session));
    }
}

这个其实就对应了Spec中的第8步，收到了来自用户的退出行为。

  1. When the runtime needs to end a running session due to the user closing or switching the application, the runtime will change the session state through appropriate intermediate ones and finally to XR_SESSION_STATE_STOPPING. When the application receives the XR_SESSION_STATE_STOPPING event, it should stop its frame loop and then call xrEndSession to tell the runtime to stop the running session.
  • 当Runtime因为收到来自用户的退出行为，或者是需要切换到其他应用时，就需要切换到XR_SESSION_STATE_STOPPING的状态
  • 当应用接收到来自Runtime的XR_SESSION_STATE_STOPPING通知时，需要停掉它的frame loop并调用xrEndSession，以告知Runtime当前Session需要stop。
  • 实际代码的话，也确实是这样，首先是xrRequestExitSession
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
XrResult
oxr_session_request_exit(struct oxr_logger *log, struct oxr_session *sess)
{
    // 1. 按照我们Demo的流程，当前session是处于focus的状态，因此依旧是is_running
	if (!is_running(sess)) {
		return oxr_error(log, XR_ERROR_SESSION_NOT_RUNNING, "Session is not running");
	}
	// 2. 处于focus，所以会进入if内，切换状态到visible,同时push一个visible事件
	if (sess->state == XR_SESSION_STATE_FOCUSED) {
		oxr_session_change_state(log, sess, XR_SESSION_STATE_VISIBLE);
	}
    // 3. 处于visible，进入到if内，切换到sync状态，push一个sync事件
	if (sess->state == XR_SESSION_STATE_VISIBLE) {
		oxr_session_change_state(log, sess, XR_SESSION_STATE_SYNCHRONIZED);
	}
    
    // 4.这里的has_ended_once在我们第一次调用xrEndFrame的时候已经设置为true了，所以不会进入到if内
	if (!sess->has_ended_once) {
		oxr_session_change_state(log, sess, XR_SESSION_STATE_SYNCHRONIZED);
		// Fake the synchronization.
		sess->has_ended_once = true;
	}
	//! @todo start fading out the app.
    // 5.切换状态到stopping，push stopping事件
	oxr_session_change_state(log, sess, XR_SESSION_STATE_STOPPING);
    // 6.设置exiting为true
	sess->exiting = true;
	return oxr_session_success_result(sess);
}
  • 因此应用端在做xrPollEvent的时候就会依次收到：XR_SESSION_STATE_VISIBLE，XR_SESSION_STATE_SYNCHRONIZED，XR_SESSION_STATE_STOPPING
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
void HandleSessionStateChangedEvent(const XrEventDataSessionStateChanged& stateChangedEvent, bool* exitRenderLoop,
                                    bool* requestRestart) {
    ......
    m_sessionState = stateChangedEvent.state;
    ......
    switch (m_sessionState) {
        ......
        // 这是之前在xr events loop中增加的注解说明：
        // 2. 当接收到STOPPING的状态后，应用端会开始做xrEndSession的操作，并置m_sessionRunning=false
        //    这样的做法，实际也是暗合了spec中第8条的说法      
        case XR_SESSION_STATE_STOPPING: {
            ......
            m_sessionRunning = false;
            CHECK_XRCMD(xrEndSession(m_session))
            break;
        }
        case XR_SESSION_STATE_EXITING: {
            *exitRenderLoop = true;
            // Do not attempt to restart because user closed this session.
            *requestRestart = false;
            break;
        }
        ......
    }
}

  • 当调用到xrEndSession，详细的Runtime进程的行为可以参考《Monaod Out of Process流程分析》一问的《Session-xrEndSession》一节。

    • 这边需要重新看一下spec的第9步，代码的部分就是这些说明描述的还原

      1. After xrEndSession, the runtime transitions the session state to XR_SESSION_STATE_IDLE. If the XR session is temporarily paused in the background, the runtime will keep the session state at XR_SESSION_STATE_IDLE and later transition the session state back to XR_SESSION_STATE_READY when the XR session is resumed. If the runtime determines that its use of this XR session has concluded, it will transition the session state from XR_SESSION_STATE_IDLE to XR_SESSION_STATE_EXITING.
      • 当应用端调用了xrEndSession之后，Runtime会把当前Session的状态变更为XR_SESSION_STATE_IDLE。如果只是临时的pasue，那么后面Runtime会继续保持Session为XR_SESSION_STATE_IDLE的状态，并在适当时候重新切到XR_SESSION_STATE_READY的状态。
      • 如果Runtime认为当前的Session已经是完全结束了，那么就会把它的状态从XR_SESSION_STATE_IDLE切换到XR_SESSION_STATE_EXITING
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
XrResult
oxr_session_end(struct oxr_logger *log, struct oxr_session *sess)
{
	......
	if (xc != NULL) {
		......
        // 1. RPC调用，会跟Runtime进程进行IPC
		CALL_CHK(xrt_comp_end_session(xc));
	}
	// 2. 首先会先把session的状态切换到idle，这里其实是暗合了spec的第9步
	oxr_session_change_state(log, sess, XR_SESSION_STATE_IDLE);
    
    // 3. 如之前的调用oxr_session_request_exit，所以sess->exiting为true，因此会进入到exiting的状态，完全结束
	if (sess->exiting) {
		oxr_session_change_state(log, sess, XR_SESSION_STATE_EXITING);
	} else {
        // 4. 如果没有exiting的操作，会直接切换到ready的状态，也是按照spec第9步
		oxr_session_change_state(log, sess, XR_SESSION_STATE_READY);
	}
	sess->has_begun = false;
	return oxr_session_success_result(sess);
}
  • 那么，很自然的之后的XR_SESSION_STATE_IDLE，XR_SESSION_STATE_EXITING，就马上在应用端做完了。
    • 因为此时整个调用的链路在events loop中，应用poll的事件还没有完全poll完，所以这一连串的session state change就在events loop中直接消化了。
1
2
3
4
5
6
case XR_SESSION_STATE_EXITING: {
    *exitRenderLoop = true;
    // Do not attempt to restart because user closed this session.
    *requestRestart = false;
    break;
}

写到这里，整个HelloXR的流程就走完了，这个sample code中其实并没有对instance做处理，因此整个流程上还是有缺的。Session End以后也不会有机会重新走到Create的流程，所以，当你退出的那一刻，这个demo其实就走完了。

Flow Chart

文字的代码实现虽然很行云流水，但是没有实际看过工程的人还是会一头雾水，另外有关多个普通Session启动的时候在哪里会skip掉，或者是block在SYNC上的细节，这部分的实现可以参考:ipc_server_process.c::ipc_server_activate_session，其中也包含了Overlay Extensions的Session VISIBLE/FOCUS状态切换的实现。

正如上述所说的，文字的部分容易让人一头雾水，所以这里补一张HelloXR的Session时序图来做一个整体描述。

image-20220622103717634

sequenceDiagram 
autonumber

rect rgb(191, 223, 255)
HelloXR ->>+ RuntimeClient: xrCreateSession
RuntimeClient -->> EventsList: push IDLE
note over RuntimeClient: change to IDLE
RuntimeClient --> EventsList: push READY
note over RuntimeClient: change to READY
RuntimeClient --> RuntimeClient: session is running
RuntimeClient -->>- HelloXR: xrCreateSession Done

HelloXR ->>+ RuntimeClient: pop Event
RuntimeClient --> RuntimeServer: RPC
RuntimeClient -->>- HelloXR: get IDLE

HelloXR ->>+ RuntimeClient: pop Event
RuntimeClient --> RuntimeServer: RPC
RuntimeClient -->>- HelloXR: get READY

end #rect rgb(191, 223, 255)

rect rgb(255, 127, 127)
HelloXR ->>+ RuntimeClient: xrBeginSession
RuntimeClient -->> RuntimeServer: RPC
RuntimeClient -->>- HelloXR: session running

HelloXR ->>+ RuntimeClient: xrWaitFrame
RuntimeClient ->>+ RuntimeServer: RPC:xrt_comp_wait_frame
RuntimeServer -->> EventsList: XRT_COMPOSITOR
_EVENT_STATE_CHANGE
note over EventsList : visible & focus is true
RuntimeServer -->>- RuntimeClient: RPC Done
RuntimeClient -->>- HelloXR: xrWaitFrame Done

HelloXR ->>+ RuntimeClient: xrBeginFrame
RuntimeClient --> RuntimeServer: RPC
RuntimeClient -->>- HelloXR: xrBeginFrame Done

HelloXR ->>+ RuntimeClient: xrEndFrame
RuntimeClient -->> EventsList: push SYNC
note over RuntimeClient: change to SYNC
RuntimeClient --> RuntimeServer: RPC
RuntimeClient -->>- HelloXR: xrEndFrame Done

HelloXR ->>+ RuntimeClient: pop Event
RuntimeClient ->>+ RuntimeServer: RPC:poll_events
RuntimeServer -->>- RuntimeClient: XRT_COMPOSITOR
_EVENT_STATE_CHANGE
RuntimeClient -->> EventsList : push visible
note over RuntimeClient: change to VISIBLE
RuntimeClient -->> EventsList : push focus
note over RuntimeClient: change to FOCUS
RuntimeClient -->>- HelloXR: get SYNC
end #rect rgb(255, 127, 127)
rect rgb(191, 223, 0)
HelloXR ->>+ RuntimeClient: pop Event
RuntimeClient --> RuntimeServer: RPC
RuntimeClient -->>- HelloXR: get VISIBLE

HelloXR ->>+ RuntimeClient: pop Event
RuntimeClient --> RuntimeServer: RPC
RuntimeClient -->>- HelloXR: get FOCUS

note over HelloXR,RuntimeClient: Frame Loop
......
xrWaitFrame
xrBeginFrame
xrEndFrame
end #rect rgb(191, 223, 0)

rect rgb(191, 223, 255)
HelloXR -->>+ RuntimeClient:xrRequestExitSession
RuntimeClient --> EventsList: push VISIBLE
note over RuntimeClient: change to VISIBLE
RuntimeClient --> EventsList: push SYNC
note over RuntimeClient: change to SYNC
RuntimeClient --> EventsList: push STOPPING
note over RuntimeClient: change to STOPPING
RuntimeClient --> RuntimeClient : session is exiting
RuntimeClient -->>- HelloXR: xrRequestExitSession Done

HelloXR ->>+ RuntimeClient: pop Event
RuntimeClient --> RuntimeServer: RPC
RuntimeClient -->>- HelloXR: get VISIBLE

HelloXR ->>+ RuntimeClient: pop Event
RuntimeClient --> RuntimeServer: RPC
RuntimeClient -->>- HelloXR: get SYNC

HelloXR ->>+ RuntimeClient: pop Event
RuntimeClient --> RuntimeServer: RPC
RuntimeClient -->>- HelloXR: get STOPPING

HelloXR ->>+ RuntimeClient: xrEndSession
RuntimeClient --> RuntimeServer: RPC
RuntimeClient --> EventsList: push EXITING
note over RuntimeClient: change to EXITING
RuntimeClient -->>- HelloXR: xrEndSession Done

HelloXR ->>+ RuntimeClient: pop Event
RuntimeClient --> RuntimeServer: RPC
RuntimeClient -->>- HelloXR: get EXITING
end #rect rgb(191, 223, 255)

总结

通过整个flow的代码走查，在Monado的实现下，其实状态是存在绑定的，比如：

  • 在xrCreateSession的时候会一次性发出IDLE和READY的状态
  • 在Runtime进程端发现Session Ready以后，会同时设置visible和focuse为ture，从而导致VISIBLE和FOCUS也是一次性发出
    • 从而不会出现spec中step 7的情况，即visible为true，focus为false
  • 在收到用户的退出行为后，整个状态机的流转事件是直接：visible，sync，idle/stopping，一下子出来

这个部分在后续实现中需要结合来看看，如果是多普通session并存（非overlay方案）的话，就需要去修改这里面的状态机了。

TODO

[ X ] Compositor Thread中对于Session的绘制管理细节

[ X ] Input / Action事件在Session中的消费/处理